The present invention relates to image processing methods, image processing apparatus, and data storage media and, more particularly, to control of the number of bits generated in a coding process of coding video signals which comprise object image data corresponding to a sequence of plural objects (i.e., plural constituents of an image).
In order to store or transmit digital image information efficiently, the digital image information should be compressively coded. At present, as methods for compressively coding the digital image information, there are Discrete Cosine Transform such as JPEG (Joint Photographic Coding Experts Group) and MPEG (Moving Picture Experts Group), and waveform coding methods such as sub-band coding, wavelet coding and fractal coding.
In addition, as an method for removing redundant image information between screens such as adjacent frames, there is a method which performs inter-frame prediction using motion compensation, i.e., represents pixel values of pixels constituting the present frame by using differential values between those pixel values and pixel values of pixels constituting its previous frame, and subjects differential image signals comprising the differential values to the waveform coding.
In recent years, in order to improve compression efficiency for video signals and at the same time reproduce the video signals in units of individual objects (constituents) constituting a prescribed image, a coding method which compressively codes object image data corresponding to the objects independently, object by object, and transmits the data is contrived. The video signals which are coded according to this coding method are subjected to a decoding processing corresponding to this coding method at the reproduction end. To be specific, in this decoding process, object coded data corresponding to each object is decoded and then respective object decoded data corresponding to the objects which are obtained by the decoding process are composed, whereby reproduced data is generated. Accordingly, an image comprising the individual objects is displayed on the basis of the reproduced data.
When the coding method which codes video signals in object units is utilized, composed images can be generated by combining the individual objects freely at the reproduction end, whereby moving pictures can be easily reedited. Further, moving pictures comprising only more significant objects can be displayed according to busyness of channels, performance of a reproduction apparatus or tastes of viewers, without reproducing relatively less significant objects.
This coding method which codes video signals in object units is called xe2x80x9cobject coding methodxe2x80x9d and this method is internationally standardized as MPEG4.
As for coded data which are obtained by compressing video signals according to the coding process as described above, an amount of codes to be decoded (the number of bits to be processed) for each prescribed time is changed. Accordingly, in order to transmit this coded data to the receiving end at a fixed bit rate and reproduce the coded data satisfactorily at the receiving end, a buffer for containing coded data should be provided at the receiving end to absorb changes in the number of bits to be processed for each prescribed time. In this case, a range of changes in the number of bits to be processed, which can be absorbed by the buffer varies according to the size of the buffer, i.e., the maximum amount of data which can be stored in the buffer.
In the standards, the maximum value of the buffer size is decided. In a receiver having a buffer of this maximum size, coded data whose number of bits to be processed per time varies are received at the fixed bit rate and reproduced without problems.
At this time at the coding end, a transmission rate is controlled at a uniform bit rate on the basis of functional restrictions, such as the size, on the buffer of the receiving end, so that the buffer does not brim over (overflow) or is not emptied (underflow).
In the object coding method, one video signal comprises object image data corresponding to plural constituents. As examples of these constituents, there are rectangular shaped images and arbitrary shaped images (objects). For example, the arbitrary shaped images, such as images of animation characters, figures and animals can be combined with the rectangular shaped image for the background.
In conventional image transmission systems, an image decoding apparatus at the receiving end includes a buffer for absorbing changes in the number of bits to be processed for each unit time as described above, corresponding to each constituent.
To be specific, at the receiving end, object coded data corresponding to one constituent is decoded by a buffer and a decoder for this constituent. Then, when video signals are coded at the transmission end, the rate control, i.e., control of the amount of codes generated as coded data is executed so that the buffer for each constituent does not overflow or underflow. In other words, the rate control in the coding process is executed independently for each constituent.
FIG. 13 is a block diagram illustrating structures of a coding apparatus and a decoding apparatus, which constitute a prior art image transmission system. Here, for convenience"" sake, a case where three constituents are processed is described. However, the number of constituents is not limited to three.
This image transmission system 1000 includes a coding apparatus 1000a for receiving image data (object image data) 1304, 1314 and 1324 which correspond to first, second and third objects (constituents), subjecting these data to compressive coding and multiplexing, and outputting a multiple bitstream 1308, and a decoding apparatus 1000b for receiving the multiple bitstream 1308, subjecting compressed data (object coded data) corresponding to the respective objects to decompressive decoding and composition, and generating reproduced data corresponding to an image which comprises the respective objects.
The coding apparatus 1000a includes first, second and third encoders 1302, 1312 and 1322 for coding the image data 1304, 1314 and 1324 which correspond to the first, second and third objects (constituents), and outputting compressed data 1305, 1315 and 1325, and a multiplexer 1307 for multiplexing the compressed data 1305, 1315 and 1325 corresponding to the respective objects, and outputting the multiple bitstream 1308. The coding apparatus 1000a further includes rate controllers 1303, 1313 and 1323 which are provided for the respective encoders 1304, 1314 and 1324 and which control amounts of data (the numbers of bits) to be decoded for each prescribed time, of the compressed data 1305, 1315 and 1325 generated by the respective encoders, on the basis of the compressed data output by the respective encoders.
The decoding apparatus 1000b includes a separator 1318 for receiving the multiple bitstream 1308 as input data 1317, and extracting compressed data 1330, 1335 and 1340 corresponding to the respective objects from the input data 1317, and first, second and third buffers 1331, 1336 and 1341 which are provided for the respective objects and contain the compressed data 1330, 1335 and 1340, respectively. The decoding apparatus 1000b further includes first, second and third decoders 1333, 1338 and 1343 which are provided for the respective objects and read compressed data 1332, 1337 and 1342 stored in the respective buffers and subjecting the read data to decoding, and a composer 1319 for composing decoded data 1334, 1339 and 1344 which are output by the respective decoders, and outputting reproduced data 1320 corresponding to a prescribed image.
In FIG. 13, the first, second and third image data are input to input terminals 1301, 1311 and 1321 of the coding apparatus 1000a, respectively. The multiple bitstream 1308 is output from an output terminal 1309 of the coding apparatus 1000a. The multiple bitstream 1308 is input to an input terminal 1310 of the decoding apparatus 1000b. The reproduced data 1320 is output from an output terminal 1327 of the decoding apparatus 1000b. 
In this image transmission system, the rate controllers 1303, 1313 and 1323 execute the rate control independently for the respective encoders 1302, 1312 and 1322 so that the buffers 1331, 1336 and 1341 at the receiving end do not overflow or underflow.
Next, the operation will be described.
In the image transmission system 1000, when the image data 1304, 1314 and 1324 corresponding to the respective objects are input to the respective input terminals 1301, 1311 and 1321, for example in the first encoder 1320, a coding process for the image data 1304 corresponding to the first object is performed in accordance with a control signal 1306 from the corresponding rate controller 1303. Also in the second and third encoders 1312 and 1322, coding processes for the image data 1314 and 1324 are similarly performed in accordance with control signals 1316 and 1326.
The compressed data 1305, 1315 and 1325 which are obtained in the coding processes by the respective encoders are multiplexed by the multiplexer 1307, and output from the output terminal 1309 as the multiple bitstream 1308.
On the other hand, in the decoding apparatus 1000b at the receiving end, when the multiple bitstream 1308 is input, the compressed data 1330, 1335 and 1340 corresponding to the respective objects are separated from the multiple bitstream 1308 by the separator 1318, and input to the corresponding buffers 1331, 1336 and 1341, respectively. The compressed data stored in the respective buffers are read by the corresponding decoders 1333, 1338 and 1343 and subjected to the decoding processes, respectively, whereby the decoded data 1334, 1339 and 1344 are generated. These decoded data 1334, 1339 and 1344 are composed by the composer 1319, and composed data is output as the reproduced data 1320 corresponding to a prescribed image.
When the coded data is transmitted from the coding apparatus to the decoding apparatus as described above, in each of the rate controllers 1303, 1313 and 1323 in the coding apparatus at the transmission end, the rate control in the coding process is executed independently for each encoder corresponding to each object, so that the buffers 1331, 1336 and 1341 at the receiving end do not overflow or underflow.
As described above, in the prior art image transmission system 1000, the respective decoders 1333, 1338 and 1343 read the compressed data from the independent buffers 1331, 1336 and 1341 and decode the data, respectively, as shown in FIG. 13. In addition, the maximum values of the sizes of the respective buffers are fixed and, in most cases, the maximum values are the same value.
In this case, at the coding end, it is possible that a value which is smaller than the maximum value of the buffer size is set as a threshold of the buffer size and the coding process is performed according to the threshold. This coding method is often used when image coded data having a smaller code amount is transmitted.
However, when the coding is performed for image data of constituents requiring large code amounts at the transmission end with executing the rate control using a larger value than the maximum value of the buffer size as the threshold, normal decoding cannot be performed by the decoders at the receiving end. In this case, in order to perform the normal decoding at the receiving end, the image data of the constituents requiring larger code amounts should be coded at a further higher compression rate. When the coding is performed at this higher compression rate, the quality of a reproduced image is deteriorated.
FIGS. 14(a)-14(c) are diagrams schematically showing by graphs transitions of occupied amounts of buffers for decoders (amounts of occupation of buffer capacities by compressed data). FIG. 14(a) shows the occupied amount of the buffer for a decoder for the first constituent. FIGS. 14(b) and 14(c) show the occupied amounts of the buffers for decoders for the second and third constituents, respectively. In the figures, the vertical axes 1401, 1411 and 1421 show the occupied amounts of the buffers for decoders, and the horizontal axes 1402, 1412 and 1422 show the display times xe2x80x9ctxe2x80x9d for the compressed data. The display times t0, t1, t2, t3, . . . show, for example, times when n-th, (n+1)-th, (n+2)-th, (n+3)-th, . . . frames are displayed.
The inclination of an inclined part 1404 of the graph in FIG. 14(a) shows the amount of codes which are input for each unit time (input bit rate) of the compressed data corresponding to the first object which is input to the buffer. In FIG. 14(a), this input bit rate is fixed. In the figure, Dm1 shows the maximum value of the compressed data which can be stored in the buffer. In any case, the buffer cannot contain the compressed data more than this maximum value.
In addition, the length of a vertical line part 1405 which is vertical to the horizontal axis in the graph shows the number of bits of the compressed data of the first object corresponding to a frame which is displayed at time t0. To be specific, bit strings are input to the buffer at the input bit rate shown by the inclination of the inclined part 1404 in the graph, and the amount of data stored in the buffer reaches a level corresponding to the peak of the vertical line part 1405 at time t0. At this time, the decoder reads the compressed data of the number of bits which is shown by the length of the vertical line part 1405, and decodes the compressed data corresponding to the frame of the time t0.
The inclinations of inclined parts 1414 and 1424 of graphs in FIGS. 14(b) and 14(c) show the amounts of codes which are input for each unit time (input bit rates) of compressed data corresponding to the second and third objects which are input to the buffers. In FIGS. 14(b) and 14(c), the input bit rates are fixed. In the figures, Dm2 and Dm3 show the maximum values of data which can be stored in the buffers 1336 and 1341, respectively. In any case, the buffers cannot contain data more than the maximum values.
The lengths of vertical line parts 1415 and 1425 which are vertical to the horizontal axes in the graphs show the numbers of bits of the compressed data of the second and third objects corresponding to frames which are displayed at time t0. To be specific, bit strings are input to the buffers at the input bit rates shown by the inclinations of the inclined parts 1414 and 1424 in the graphs, and the amounts of data stored in the buffers reach levels corresponding to peaks of the vertical line parts 1415 and 1425 at time t0. At this time, the decoders read the compressed data of the numbers of bits which are shown by the lengths of the vertical line parts 1415 and 1425, and decodes the compressed data corresponding to the frames of time t0.
Here, assume that the first constituent (object) is a constituent which requires a large amount of codes in the coding process and the second constituent (object) is a constituent which requires a small amount of codes in the coding process. The buffer for the first constituent (FIG. 14(a)) is occupied by the compressed data up to approximately the maximum value of the data storage amount. On the other hand, the buffer for the second constituent (FIG. 14(b)) has still a considerable margin in the data storage amount.
In this case, in the prior art image transmission system, a data storage area in the buffer having a small amount of stored data cannot be allocated to the compressed data for the first constituent. Accordingly, the image data for the first constituent should be compressed at a higher compression rate than that of the second constituent, whereby the quality of the image of the first constituent is deteriorated.
Further, as for the third constituent (FIG. 14(c)), compressed data which are read at times t0 and t1 have small code amounts and compressed data which is read at time t2 has a large code amount. In a buffer corresponding to a constituent, the complexities of its images being largely changed with a lapse of time, its data storage area is almost all occupied. As for this constituent, the allocation of the code amount to the compressed data is restricted in a frame in which the compressed data is read at time t2 by the restriction on the buffer size, whereby the quality of the image for this frame is deteriorated accordingly.
In addition, in some cases, even when the complexities of all constituents or changes in the complexities are the same on the average, an image of a specific constituent is locally more complex than images of other constituents at a certain time. In these cases, even when a larger number of bits are to be allocated to data corresponding to the specific constituent to code the data, the number of bits can be allocated only up to the maximum limit of the data storage amount of the corresponding buffer for the specific constituent due to the restriction of the buffer size, even when there are margins in the amounts of occupation of buffers by data corresponding to other constituents. Accordingly, when the image of the specific constituent is complex, the quality of the image cannot be improved.
As described above, in the prior art, the buffer for containing coded data (compressed data) is provided independently for each constituent (i.e., each decoder) at the receiving end. Therefore, the buffers cannot be effectively utilized among the constituents, whereby the quality of an image of a specific constituent is deteriorated and the quality of a composed image becomes uneven.
Further, transmission rates of compressed data for the respective objects are previously set in consideration of the complexities of images of the objects. Therefore, the transmission rates are not always suitable for actual communication lines having limited data transmission speeds.
It is an object of the present invention to provide an image processing method and image processing apparatus, which can utilize buffers for containing coded data effectively among plural different constituents, and kinetically distribute the number of bits according to the complexities of the respective constituents unless the buffers do not overflow or underflow, and a data storage medium which contains a program for executing the image processing according to the image processing method by a computer.
It is an object of the present invention to provide an image processing method and image processing apparatus, which can adapt transmission rates of compressed data corresponding to objects to actual communication lines having limited data transmission speeds, and a data storage medium which contains a program for executing the image processing according to the image processing method by a computer.
Other objects and advantages of the present invention will become apparent from the detailed description and specific embodiments described are provided only for illustration since various additions and modifications within the spirit and scope of the invention will be apparent to those of skill in the art from the detailed description.
An image processing method according to a 1st aspect of the present invention, of subjecting a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, comprises: a complexity measurement step of measuring complexity of each constituent, corresponding to a number of bits required for the coding processing for each of the object image data; and a number-of-bits-allocation-ratio decision step of deciding a ratio of a number of bits which are allocated to each of the object image data among the plural constituents, according to the complexity of each constituent, and each of the object image data is coded so that a number of bits for each of the object coded data meets the decided ratio of the number of bits allocated among the constituents.
According to a 2nd aspect of the present invention, an image processing method of decoding multiplex coded data obtained by multiplexing object coded data for constituents, which are obtained according to the image processing method of the 1st aspect, comprises: a separation step of separating the object coded data for each of the constituents from the multiplex coded data; a storage step of storing the object coded data for each of the constituents in a buffer; and a decoding step of extracting the object coded data for each constituent from the buffer and decoding the extracted object coded data.
According to a 3rd aspect of the present invention, in the image processing method of the 1st aspect, an index indicating the complexity of each of the constituents is one of a first value which indicates a variance of pixel values of pixels constituting each of the constituents, a second value which indicates a magnitude of a temporal change of a display image of each of the constituents, a third value which is obtained by weighting a number of bits generated in coding a corresponding constituent in a previously processed frame which was subjected to the coding processing before a target frame as a target of the coding processing, by a ratio of a coefficient indicating characteristics of the constituent between the target frame and the previously processed frame, and a fourth value which indicates a number of generated bits for each of the constituent, which are generated in preliminarily coding the object image data for each of the constituents under prescribed coding conditions.
An image processing method according to a 4th aspect of the present invention, of subjecting a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, frame by frame as a unit of a display process, comprises: a number-of-bits-per-frame decision step of deciding a number of bits per frame, which are allocated to a video signal corresponding to a target frame as a target of the coding processing; and a number-of-bits distribution step of distributing the number of bits per frame so as to correspond to each of constituents of the target frame, according to complexity of each of the constituents, to decide a number of bits per object for each of the constituents corresponding to the target frame, and each of the object image data corresponding to the target frame is coded so that a number of bits of corresponding object coded data is equal to the number of bits per object of each of the constituents corresponding to the target frame.
An image processing method according to a 5th aspect of the present invention, of subjecting a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, frame by frame as a unit of a display process, comprises: a number-of-bits-per-frame decision step of deciding a number of bits per frame, which are allocated to a video signal corresponding to a target frame as a target of the coding processing; and a number-of-bits-allocation-ratio decision step of deciding a ratio of a number of bits which are allocated to each of the object image data among plural constituents of the target frame, according to a ratio of complexity of each of the constituents to a sum of complexities of the plural constituents, and each of the object image data is coded so that the number of bits per frame is distributed to each of the object coded data according to the decided ratio of the number of bits which are allocated among the plural constituents.
An image processing apparatus according to a 6th aspect of the present invention, which subjects a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, comprises: a complexity measurement unit for measuring complexity of each of the constituents, which corresponds to a number of bits required for the coding processing for each of the object image data; a number-of-bits allocation unit for allocating a number of bits which are used in coding object image data, to each of constituents of a target frame as a target of the coding processing, according to the complexity of each of the constituents; an encoder for coding each of object image data corresponding to the target frame, on the basis of the number of bits allocated to each of the constituents, to generate the object coded data; and a multiplexer for multiplexing the object coded data for the constituents, to generates a bitstream.
According to a 7th aspect of the present invention, in the image processing apparatus of the 6th aspect, the number-of-bits allocation unit allocates the number of bits to each of the constituents on the basis of a ratio of the complexity of each of the constituents to a sum of the complexities of all the constituents of the target frame.
According to an 8th aspect of the present invention, in the image processing apparatus of the 6th aspect, the complexity measurement unit measures the complexity of each of the constituents using as an index, one of a first value which indicates a variance of pixel values of pixels constituting each of the constituents, a second value which indicates a magnitude of a temporal change of a display image of each of the constituents, a third value which is obtained by weighting a number of bits generated in coding a corresponding constituent in a previously coded frame which was subjected to the coding processing before the target frame, by a ratio of a coefficient indicating characteristics of the constituent between the target frame and the previously processed frame, and a fourth value which indicates a number of generated bits for each of the constituent, which are generated in preliminarily coding the object image data for each of the constituents under prescribed coding conditions.
According to a 9th aspect of the present invention, in the image processing apparatus of the 6th aspect, the number-of-bits allocation unit distributes a number of bits per frame, which are allocated to a video signal of the target frame, to each of the constituents, so that a buffer for containing the object coded data corresponding to each of the constituents, which data is separated from the bitstream in an image decoding apparatus which subjects the bitstream to a decoding processing does not overflow or underflow.
According to a 10th aspect of the present invention, there is provided a data storage medium which contains an image processing program in which the image processing program is a program for executing the coding processing for video signals according to the image processing method of the 1st aspect, by a computer.
According to an 11th aspect of the present invention, there is provided an image processing method which includes a data processing at transmission end for converting a video signal for displaying a prescribed image into a bitstream, frame by frame, according to a coding processing, and transmitting the bitstream, and a data processing at receiving end for receiving the bitstream, and converting the bitstream into reproduced data, frame by frame, according to a decoding processing, in which the data processing at transmission end comprises: an object coding step of coding object image data of a target frame as a target of the coding processing, corresponding to plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data; and a multiplexing step of multiplexing the object coded data corresponding to the constituents, to generate the bitstream, and the data processing at receiving end comprises steps of: a data storage step of storing the bitstream in one buffer, frame by frame; a decoding step of reading a bitstream corresponding to the target frame from the buffer and decoding the read bitstream separately for each constituent, to generate reproduced data corresponding to each of the constituents; and a composition and display step of composing the reproduced data corresponding to the constituents to generate composed data, and displaying the prescribed image on the basis of the composed data.
According to a 12 aspect of the present invention, in the image processing method of the 11th aspect, the object coding step comprises: a number-of-bits decision step of deciding a number of bits per frame, which are allocated to the target frame as a target of the coding, so that the buffer does not underflow or overflow; and a number-of-bits distribution step of distributing the number of bit per frame so as to correspond to each of constituents of the target frame according to complexity of each of the constituents, to decide a number of bits per object for each of the constituents, and the object image data for each of the constituents of the target frame is coded on the basis of each of the number of bits per object.
According to the 1st to 12th aspects of the present invention, the coding processing for the image data of each of the constituents can be performed on optimal conditions, whereby the quality of a whole reproduced image of a video signal can be increased.
Especially according to the 3rd or 4th aspect of the present invention, the decoding processing at the receiving end can always be performed satisfactorily.
An image processing apparatus according to a 13th aspect of the present invention, which decodes multiplex coded data which is obtained by subjecting a video signal for displaying a prescribed image to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent to convert the object image data into object coded data, and multiplexing the object coded data, comprises: a separator for separating the object coded data corresponding to each of the constituents from the multiplex coded data; a buffer for containing the object coded data corresponding to each of the constituents; and a decoder for extracting the object coded data corresponding to each of the constituents from the buffer and decoding the object coded data. Therefore, the data storage area of the buffer can be commonly used effectively among the plural constituents. That is, the change in the number of bits allocated to each constituent according to the complexity of the constituent can be absorbed among the plural constituents. Consequently, an optimal number of bits can be allocated to the image of each constituent, whereby the quality of the whole image can be increase.
An image processing method according to a 14th aspect of the present invention, for subjecting a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, frame by frame as a unit of a display process, comprises: a complexity obtaining step of obtaining a number of generated bits for each of the constituents, which bits are obtained by preliminarily coding the object image data corresponding to each of the constituents, as an index of complexity of each of the constituents; a distribution-ratio-per-frame obtaining step of obtaining a ratio of the complexity of a target frame as a target of a main coding processing, to a sum of the complexities of all frames which are not subjected to the main coding processing yet, as a distribution ratio per frame; a number-of-bits-per-frame decision step of multiplying a number of bits which are not used yet among a total number of bits which are allocated to the video signal, by the distribution ratio per frame, to decide a number of bits which are allocated to the target frame; a distribution ratio obtaining step of obtaining a ratio of complexity of each of constituents of the target frame as a target constituent of the main coding processing to a sum of complexities of all the constituents, as a distribution ratio per constituent; a number-of-bits-per-constituent decision step of multiplying the number of bits which are allocated to the target frame, by the distribution ratio per constituent, to decide a number of bits which are allocated to the target constituent; and a constituent coding step of coding the object image data corresponding to each of the constituents of the target frame separately on the basis of the number of bits allocated to each of the constituents.
An image processing apparatus according to a 15th aspect of the present invention, which subjects a video signal for displaying a prescribed image, to an object image processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, comprises: a complexity measurement unit for obtaining a number of generated bits for each of the constituents, which bits are generated by preliminary coding the object image data corresponding to each of the constituents, as an index of complexity of each of the constituents; a distribution-ratio-per-frame obtaining unit for obtaining a ratio of the complexity of a target frame as a target of a main coding processing, to a sum of the complexities of all frames which are not subjected to the main coding processing yet, as a distribution ratio per frame; a number-of-bits-per-frame decision unit for multiplying a number of bits which are not used yet among a total number of bits which are allocated to the video signal, by the distribution ratio per frame, to decide the number of bits which are allocated to the target frame; a distribution ratio obtaining unit for obtaining a ratio of complexity of each of constituents of the target frame as a target constituent of the main coding processing to a sum of complexities of all the constituents, as a distribution ratio per constituent; a number-of-bits-per-constituent decision unit for multiplying the number of bits which are allocated to the target frame, by the distribution ratio per constituent, to decide a number of bits which are allocated to the target constituent; a coding unit for coding each of the constituents of the target frame separately on the basis of the number of bits allocated to each of the constituents, to generate the object coded data; and a multiplexer for multiplexing the object coded data corresponding to the plural constituents, and outputting a bitstream.
According to a 16th aspect of the present invention, there is provided a data storage medium which contains an image processing program, in which the image processing program is a program for executing the coding processing of video signals according to the image processing method of the 14th aspect, by a computer.
According to a 17th aspect of the present invention, there is provided an image processing method of subjecting a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, in which the object coding processing comprises: a complexity measurement step of measuring complexity of each of the constituents, which corresponds to a number of bits required for the coding processing for each of the object image data; and a rate ratio decision step of deciding a ratio of a bit rate of the object coded data for each of the constituents among the plural constituents, according to the complexity of each of the constituents, and the object image data for each of the constituents is coded at a prescribed bit rate on the basis of the ratio of the bit rate for each of the constituents, so that a sum of the bit rates for all the constituents has a fixed value.
An image processing method according to an 18th aspect of the present invention, of decoding multiplex coded data obtained by multiplexing the object coded data for the constituents, which are obtained according to the image processing method of the 17th aspect comprises: a separation step of separating the object coded data for each of the constituents from the multiplex coded data; a storage step of storing the object coded data for each of the constituents in one buffer; and a decoding step of extracting the object coded data for each of the constituents from the buffer, and decoding the object coded data.
According to a 19th aspect of the present invention, in the image processing method of the 17th aspect, the object coding processing comprises: a coding condition decision step of deciding a ratio of a bit rate for each of the constituents and a maximum amount of the object coded data stored in the buffer, corresponding to each of the constituents, according to the complexity of each of the constituents, so that the buffer does not underflow or overflow, and the object image data for each of the constituents is coded at a prescribed bit rate on the basis of the ratio of the bit rate for each of the constituents, so that a sum of the bit rates for all the constituents has a fixed value.
According to a 20th aspect of the present invention, in the image processing method of the 17th aspect, an index indicating the complexity of each of the constituents is one of a first value which indicates a variance of pixel values of pixels constituting each of the constituents, a second value which indicates a magnitude of a temporal change of a display image of each of the constituents, a third value which is obtained by weighting a number of bits generated in coding a corresponding constituent in a previously processed frame which was subjected to the coding processing before a target frame as a target of the coding processing, by a ratio of a coefficient indicating characteristics of the constituent between the target frame and the previously processed frame, and a fourth value which indicates a number of generated bits for each of the constituents, which are generated in preliminarily coding the object image data for each of the constituents under prescribed coding conditions.
According to a 21st aspect of the present invention, there is provided an image processing method of subjecting a video signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, in which the object coding processing comprises: a complexity measurement step of measuring complexity of each of the constituents, which corresponds to a number of bits required for the coding processing of each of the object image data; and a coding condition decision step of deciding a ratio of a bit rate of the object coded data for each of the constituents among the plural constituents, and a maximum amount of the object coded data stored in the buffer at a decoding end, corresponding to each of the constituents, according to a ratio of the complexity of each of the constituents to a sum of the complexities of all the constituents, and the object image data for each of the constituents is coded at a prescribed bit rate on the basis of the ratio of the bit rate for each of the constituent, so that a sum of the bit rates for all the constituents has a fixed value.
An image processing apparatus according to a 22nd aspect of the present invention, which subjects a vide signal for displaying a prescribed image, to an object coding processing of coding object image data for plural constituents of the prescribed image separately for each constituent, to convert the object image data into object coded data, comprises: a complexity measurement unit for measuring complexity of each of the constituents, which corresponds to a number of bits required for the coding processing for each of the object image data; a coding condition decision unit for deciding an individual bit rate for each of the constituents and an individual buffer size for each of the constituents as a maximum amount of the object coded data stored in a buffer at a decoding end, according to the complexity of each of the constituents; a controller for outputting a control signal on the basis of the decided individual bit rate and individual buffer size; an encoder for performing the coding processing for the object image data for each of the constituent, with controlling a number of bits generated by the coding processing on the basis of the control signal, and outputting the objet coded data for each of the constituents; and a multiplexer for multiplexing the object coded data for the constituents, to generate a bitstream.
According to a 23rd aspect of the present invention, in the image processing apparatus of the 22nd aspect, the coding condition decision unit distributes a maximum transmission bit rate of a transmission path to each of the constituents according to the complexity of each of the constituents, to decide the individual bit rate for each of the constituents, and distributes a maximum data storage amount of the buffer at the decoding end to each of the constituents according to the complexity of each of the constituents, to decide the individual buffer size.
According to a 24th aspect of the present invention, there is provided a data storage medium which contains an image processing program, in which the image processing program is a program for executing the coding processing for video signals according to the image processing method of the 17th aspect, by a computer.
According to the 17th to 24th aspects of the present invention, the transmission rate of compressed data (coded data) for an object can be adapted to an actual communication line having a restricted data transmission speed.